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The  last  several  years  have  brought  about  changes  regarding  how  test  and  evaluation  will  he 
done,  moving  the  focus  from  the  individual  system  level  up  to  mission  impacts  and  the  value 
added  to  current  capabilities.  As  a  result,  the  Army  Evaluation  Center  is  making  changes  to 
include  the  implementation  of  a  mission-based  test  and  evaluation  methodology  that  focuses  on  a 
system’s  contribution  to  the  overall  mission.  This  new  methodology  has  been  documented  and  is 
currently  being  executed  but  has  proven  to  be  rather  cumbersome  and  difficult  to  implement.  To 
overcome  the  challenges  and  move  evaluators  from  Word  documents  and  Excel  sheets  into  the 
technology  age  of  user  forms  and  online  accessibility,  we  have  developed  a  system  concept  called 
the  Test  and  Evaluation  Support  Tool  and  Example  Repository  (TESTER). 
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In  the  Army  Evaluation  Center  (AEC),  a 
subordinate  command  of  U.S.  Army  Test 
and  Evaluation  Command  (ATEC),  the 
mission  is  to  provide  independent  evaluation 
of  systems  through  the  use  of  Test  and 
Evaluation  (T8cE)  to  provide  decision  makers  with 
information  on  system  performance  and  the  ability  of 
the  system  to  meet  customer  needs.  It  only  makes  sense 
then,  that  AEC  should  use  systems  engineering,  the 
processes  beginning  with  the  identification  of  customer 
needs  and  continues  throughout  the  acquisition  life 
cycle  in  every  facet  of  daily  business. 

In  reality,  however,  evaluations  often  play  a  game  of 
catch-up  and  start  later  in  the  system  development  life 
cycle  than  ideal.  The  result  includes  minimal  evaluation 
and  understanding  of  how  a  system  will  affect  higher- 
level  missions  and  limited  ability  to  show  improved 
functionality  and  value  of  deployed  units  with  the 
additional  systems.  Current  limitations  of  testing  and 
evaluating  systems  resulted  in  policy  changes  for  T8cE 
recorded  in  the  Department  of  Defense  Instruction 
5000.02  dated  December  8,  2008  (DoD,  2008). 
Changes  included  using  TScE  to  “assess  improvements 
to  mission  capability. . .  based  on  user  needs  and  should 
be  reported  in  terms  of  operational  significance  to  the 
user”  and  requiring  evaluations  to  “include  a  comparison 
with  current  mission  capabilities.” 


Evaluation  transformation 

To  meet  the  changes  laid  out  in  the  DoDI  5000.02, 
AEC  has  begun  implementing  some  changes  in  how 
T8dE  is  approached  and  executed.  One  of  the  key 
changes  has  been  the  implementation  of  a  Mission-Based 
Test  and  Evaluation  (MBTScE)  process,  which  takes  the 
evaluator  through  a  step-by-step  process,  ensuring  the 
system  is  evaluated  based  on  its  contribution  to  the 
mission  as  a  whole  and  value  added  to  current  capabi¬ 
lities.  An  article  addressing  these  poUcy  changes  and  the 
MBT8cE  initiative  was  published  in  the  September  2009 
ITEIA  Journal,  which  provides  an  outline  of  the  process 
and  purpose  of  MBTScE  (ApiceUa,  2009). 

MBTScE  is  stiU  in  its  infancy  for  use  within  AEC, 
having  just  been  initiated  in  January  2009  and,  as  a 
result,  lessons  are  currently  being  learned  and  recorded 
to  improve  the  process.  One  of  the  key  lessons  that  has 
been  captured  thus  far  is  the  realization  that  the 
process  is  rather  cumbersome  and  the  spreadsheet 
designed  to  capture  evaluation  measures  and  data  can 
become  overwhelming  in  size.  To  try  and  overcome 
these  challenges  and  bring  evaluations  into  the  new 
technology  age,  we  are  developing  a  new  system  named 
the  Test  Sc  Evaluation  Support  Tool  and  Example 
Repository  (TESTER).  The  TESTER  system  will  be  a 
Web-based  program  offering  many  benefits  to  evalu¬ 
ations  including: 
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•  provide  an  online  electronic  system  for  evaluators 
and  system  stakeholders  to  work  collaboratively 
on  system  evaluations; 

•  guide  users  through  the  MBT&E  steps  using 
interactive  interfaces  (such  as  drop-down  menus) 
to  facilitate  an  easier-to-use  process; 

•  develop  standardized  formats  for  evaluation 
products  while  still  allowing  content  flexibility 
to  meet  varying  system  evaluation  needs; 

•  link  Cross  Command  Collaboration  Effort 
(3CE)  products  into  system  evaluations; 

•  ensure  standardized  system  names  and  schedules 
by  direct  connections  with  the  ATEC  decision 
support  system;  and 

•  facilitate  storage  of  documents  on  a  digital  library 
enabling  a  common  document  repository. 

The 

TESTER 
concept 

As  trial 
mission- 
based  eval¬ 
uations  were 
being  exe¬ 
cuted,  an 

MBTScE 

working 
group  was 
formed  to 
assess  the 

needs  of  evaluators  using  this  new  methodology.  It  was 
determined  that  an  online  system  would  provide  needed 
assistance  to  users  to  facilitate  the  new  evaluation  process. 
While  existing  systems  could  be  used  to  support  various 
aspects  of  MBTScE,  some  new  development  would  stiU 
need  to  occur  to  meet  the  implementation  methods 
outlined  by  the  MBTScE  process  and  guide.  Based  on 
this,  the  TESTER  concept  was  designed  to  use  existing 
programming  and  build  new  interfaces  following  the 
MBTScE  implementation  guide.  The  concept  was  further 
refined  with  lessons  learned  from  AEC  evaluators  using 
MBTScE  in  their  evaluations  and  future  needs  of  large 
system-of-system  programs  like  the  U.S.  Army  Brigade 
Combat  Team  Modernization  program. 

MBT&E  interface 

TESTER  wiU  consist  of  three  main  components: 
MBTScE  user  forms,  3CE  metric  catalog,  and  an 
example  repository.  The  first  component,  MBTScE  user 
forms,  wiU  walk  the  user  through  each  step  in  the 
MBTScE  process  beginning  with  the  coUection  of 
information.  This  first  step  wiU  be  a  collaborative  effort 


between  TESTER  and  the  digital  library.  The  digital 
library  wiU  facUitate  the  storage  of  documents  according 
to  their  existing  file  structure  organized  by  system. 
ATEC  system  team  members  and  other  stakeholders 
will  be  provided  links  and  guided  to  information  used  for 
the  system  evaluation,  aU  maintained  on  one  common 
site.  The  user  is  then  led  through  steps  to  capture  which 
missions  the  system  wiU  address  and  which  tasks  will  be 
performed.  Throughout  the  MBT&E  process,  users  will 
be  steered  toward  an  evaluation  that  considers  the  wider 
scope  of  how  the  system  wiU  add  value  to  current 
capabUities  and  increase  the  potential  for  a  unit  to 
accomplish  its  mission. 

During  these  steps,  TESTER  will  provide  time¬ 
saving  features  like  drop-down  menus  for  standardized 
information  and  the  selection  of  information  previously 

entered  into 
the  system. 
Other  fea¬ 
tures  wiU  in¬ 
clude  one- 
chck  selec¬ 
tions  to  hnk 
information 
between 
steps,  auto¬ 
numbering 
when  enter¬ 
ing  infor¬ 
mation,  the 
ability  to  re¬ 
arrange  the 

order  of  information,  and  autocorrecting  of  numbering 
when  rearranging  the  order  of  information.  These 
features  will  help  reduce  duphcative  entry  of  information 
and  provide  a  vast  improvement  from  current  methods 
being  used,  including  Microsoft  Word  and  Excel. 
Additionally  the  user  interface  wiU  be  intuitive,  aUowing 
users  to  interface  with  the  system  without  the  need  for 
specialized  training.  FinaUy  TESTER  wiU  produce 
standardized  documents  for  evaluations  and,  using  Unks 
to  the  digital  Ubrary,  aUow  for  easy  storage  of  fmaUzed 
evaluation  products. 

3CE  metric  catalog 

The  second  component  of  TESTER  is  the  3CE 
effort  metric  catalog.  3CE  was  initiated  in  2004 
following  the  2003  Army  Modeling  and  Simulation 
Executive  Council  meeting,  where  a  memorandum  of 
understanding  was  signed  between  the  Training  and 
Doctrine  Command,  ATEC,  and  Research,  Develop¬ 
ment  and  Engineering  Command  to  work  toward 
distributed  Modeling  and  Simulation  (M&JS).  3CE 
has  a  mission  to  develop  a  cross  command  Army  M&S 
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and  data  environment  for  design,  development,  inte¬ 
gration,  and  testing  of  capabilities,  systems,  and 
prototypes.  To  accomplish  this,  3CE  has  been  obtaining 
requirements  from  the  three  commands,  program 
managers,  and  other  stakeholders  (like  the  lead  system 
integrator)  since  2004  and  building  a  database  of 
requirements.  The  database  has  been  further  enhanced 
by  decomposing  the  requirements  for  data  elements, 
tvhich  are  the  data  and  information  that  will  need  to  be 
collected  from  a  test  or  MScS  to  answer  the  require¬ 
ment.  This  information  is  used  in  the  development  of 
system  engineering  architecture  that  shows  the  M8cS 
environment  and  data  requirements.  Additional  efforts 
are  currently  underway  to  compare  the  requirements  to 
current  M6cS  tools  to  identify  gaps  and  possible 
solutions  to  meet  requirement  needs. 

Currently,  3CE  products  reside  on  the  knowledge 
repository  located  on  the  Army  Knowledge  Online 
system;  however,  there  is  no  ability  to  view  and  search 
the  database  of  requirements  online.  TESTER  will 
provide  online  access  to  the  3CE  database,  called  the 
metrics  catalog,  which  includes  all  requirements 
collected  by  3CE,  data  elements,  calculations  to  answer 
requirements,  and  potential  data  sources.  3CE  is  also 
developing  Department  of  Defense  architecture  frame¬ 
work  views,  which  support  one  of  the  final  steps  in  the 
MBT&E  process.  The  information  and  products  being 
developed  by  3CE  will  also  be  linked  into  the 
MBT&E  user  forms  component,  providing  users  with 
a  wealth  of  information  to  draw  from  while  developing 
their  system  evaluations. 

Example  repository 

The  final  component  of  TESTER  will  provide  users 
with  examples  of  historical  reference  material  such  as 
system  evaluation  plans,  verification,  validation,  and 
accreditation  documents,  and  model  informational 
reports.  These  materials  will  be  searchable  and  export¬ 
able  for  ease  of  use  and  provide  users  with  references  for 
future  projects.  The  example  repository  will  also  provide 
additional  references  for  MBT&E  products  and  the  use 
and  implementation  of  3CE  products. 


The  future  outlook 

The  need  for  TESTER  to  facilitate  MBT&E 
processes  is  a  current  and  pressing  issue,  so  work  has 
been  swiftly  executed  to  develop  a  technical  capabilities 
requirement  document.  Concurrently  meetings  have 
been  held  with  collaborative  systems  proponents  and 
with  individuals  who  have  participated  in  the  trial 
MBT&E  evaluations  and  assessments.  Information 
and  experiences  have  been  gathered  to  ensure  user 
needs  are  met  and  that  TESTER  will  provide  an 
essential  tool  to  facilitate  system  evaluation.  TESTER 
is  expected  to  be  fuUy  developed  and  utilized  by  the 
spring  of  2011  with  efforts  underway  to  reduce  the 
development  time  to  meet  pressing  needs.  □ 
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